The big test for the stock market's Trump rally may be President Trump himself.

Wall Street's post-election surge was built on hopes of a "pro-growth" agenda getting swiftly implemented. Investors bet big that Trump's promises to slash taxes, unleash infrastructure spending and cut regulation would accelerate the sluggish American economy.

But those hopes are already getting undercut by Trump's turbulent first days in office.

Instead of focusing on pushing his stimulus plans through Congress, as investors had hoped, Trump is instead acting first on polarizing issues like the ban on travelers from seven majority-Muslim nations. The White House has also taken an aggressive stance on trade, raising concerns on Wall Street over the risk of a trade war.

In other words, investors want tax cuts and spades in the ground, not deeply divisive travel bans.

After a weekend of confusion and protests over the Trump travel ban, the Dow on Monday dropped as many as 223 points before closing down 123 points.

It's not a panic, but still qualifies as Wall Street's worst day since Trump's victory. The VIX (VIX), a gauge of market volatility, spiked by 12% -- the biggest increase in nearly three months.

"There is a palpable new feeling out there that this administration is not going to be linearly ideal for markets," Michael Block, chief market strategist at Rhino Trading, wrote in a note.

"Maybe it will be pro-business overall, but this isn't a matter of snapping one's fingers," Block said.

Jurgen Hardt, a German foreign ministry official, said he's anxiously awaiting the moment when the stock markets "turn from good to bad."

"An America that operates by itself will no longer be attractive for investors," Hardt said.

Related: Trump on Dow 20K: 'Now we have to go up, up, up

Of course, Monday's market pullback barely puts a dent in a post-election rally that had carried the Dow more than 1,700 points higher. In fact, the losses don't even wipe out last week's strong gains as the Dow blew past the 20,000 milestone.

It's only reasonable -- and healthy -- for the markets to take a breather after such a move.

It's especially apt because there are growing doubts that Trump will be able to get his stimulus plans done fast enough to support the big gains on Wall Street.

Sean Spicer, the White House press secretary, was asked during Monday's press briefing about the market decline and concerns over a delay in tax reform.

Spicer declined to commit to getting tax reform done this year, but said the administration will "continue to work with both houses of Congress."

Trump's promises to cut taxes and ramp up infrastructure spending were always going to be complex and time-consuming to deliver on, even with the GOP in charge of Congress.

The risk is that Trump's early focus on the travel ban, alleged voter fraud and other controversial issues will slow his stimulus plans by emboldening opponents and fracturing would-be allies in Congress.

Chris Krueger, a policy analyst at Cowen & Co., expressed frustration with the "surreal" start to Trump's presidency.

He said the lack of coordination between the White House and Congress doesn't bode well for "gigantic legislative deals" on Obamacare, infrastructure and tax reform.

"This is Trump's second full week as President. With his party in charge of Congress. It really shouldn't be this hard," Krueger wrote.

Related: Tech firms fighting Trump's travel ban with cash

And then there's Trump's tough trade rhetoric. Trump formally withdrew the U.S. from the TPP trade deal and wants to renegotiate the NAFTA alliance with Canada and Mexico.

The White House has also suggested paying for a wall along the border with Mexico by putting a 20% tax on Mexican imports -- even though that could risk a trade war and would hurt American consumers.

The trade rhetoric, along with new immigration policies, add to a sense that the U.S. is retreating from the global stage.

Block noted that while the market has brushed off political turmoil many times lately, "now we're dealing with a bigger issue -- the U.S. alienating the rest of the world."

Larry Summers, a senior economic official in the Obama administration, believes the markets and economy are "enjoying a sugar-high" that won't last a year.

Writing in the Washington Post's Wonkblog, Summers pointed out that some previous governments with "authoritarian tendencies," including those of Hitler and Mussolini, sparked bull markets "before they led to disaster."

"After the events of the last week, it is much easier to imagine downside than upside scenarios," Summers wrote.