The troops of General A. P. Hill's Corps resumed their former positions of July 3rd, and remained there until the night of the 4th. When the order to march was given the men of A. P. Hill's Corps would travel toward Hagerstown, by way of Fairfield and Waynesboro. During the afternoon, General Hill's Corps began marching out of Fairfield taking the same route that General Ewell's wagon supply trains took a day earlier. The road was badly torn up which made the march grind to a slow pace. It was almost 1 a.m. on July 6th, when General Hill's men approached the mountain pass near Monterey. Sergeant Gochenour recorded the march from Gettysburg. On July 4th the Danville Artillery was ordered to retreat moving roughly 5 miles within twenty-four hours. They moved up to Monterey during the night of the 5th and was ordered to deploy on a hill two and half miles from Maryland overlooking Waynesboro. They were stationed there until late July 6th and then ordered to march to Waynesboro. The Danville Artillery entered Waynesboro near 8 P.M. that evening.

On the night of July 4th, General Longstreet's Corps was ordered to march to Hagerstown following behind the troops of A. P. Hill's Corps. During the day of July 5th, General Longstreet's First Corp were struggling to cross over Jack's Mountain. Longstreet's Corps was en route on the Waynesboro Pike traveling from Fairfield. They passed through the little village of Fountain Dale reaching the summit of the mountain and succeeded in reaching the top of the mountain early that night. Longstreet's Corp had passed by Monterey Springs, on the summit of the mountain where it crossed over to Waynesboro. Shortly before midnight the first parts of Longstreet's Corps began marching toward Waterloo (modern day Rouzerville) taking the lead as it traveled to Hagerstown. His men encamped near Monterey along the Waynesboro and Emmitsburg Pike for the night.

With Longstreet’s Corps were General Pickett and Colonel Porter Alexander who were in charge of prisoner detail. There during the night the two men spoke briefly about the shattered remains of Pickett's' Division. So sad was General Pickett that he never recovered from the loss of his men during the famous charge on July 3rd known as Pickett's Charge.

During the afternoon, British observer Colonel Fremantle rejoined General Longstreet on the road to the top of the South Mountain. At 4:00 p.m., they stopped on the western slope of the mountain where the road forked, one heading to Emmitsburg, the other to Hagerstown. Near the intersection, they noticed an old farmhouse. Colonel Fremantle and Major Moses entered the farmhouse where they found several wounded Yankees, and one who had died. They had been wounded in the action of Monterey. The women in this house were great abolitionists. When Major Fairfax rode up, he asked whether the corpse was that of a Confederate or Yankee. The body was in the veranda, covered with a white sheet. The woman replied shaking her foot saying, "If there was a Confederate in here he wouldn’t be here long."

Colonel Fremantle the British observer speaks about the beginning phase of the march:

"The night was very bad thunder and lightning, torrents of rain the road knee deep in mud and water, and often blocked up with wagons "come to grief." I pitied the wretched plight of the unfortunate soldiers who were to follow us. Our progress was naturally very slow indeed, and we took eight hours to go as many miles. At 8 am. we halted a little beyond the village of Fairfield, near the entrance to a mountain pass. No sooner had we done so and lit a fire, than an alarm was spread that Yankee cavalry were upon us. Several shots flew over our heads, but we never could discover from whence they came. News also arrived of the capture of the whole of Ewell's beautiful wagons. These reports created a regular stampede amongst the wagoners, and Longstreet's drivers started off as fast as they could go. Our medical trio, however, firmly declined to budge, and came to this wise conclusion, partly urged by the pangs of hunger, and partly from the consideration that, if the Yankee cavalry did come, the crowded state of the road in our rear would prevent our escape. Soon afterwards, some Confederate cavalry were pushed to the front, who cleared the pass after a slight skirmish."

It was almost noon when General Ewell's Second Corps began moving on the Fairfield Road. General Ewell's Corps reached Fairfield by 4 P.M. The march was six to eight miles from Marsh Creek near Gettysburg to Fairfield. The Lee Battery, part of Johnson's Division followed the long road home leaving many of it's wounded men behind. The 50th Virginia Infantry had heard the news of the battle at Monterey and upon seeing the conditions around them as they camped at Fairfield for the night made them wonder if the battle of Gettysburg was worth the fight. Adding to the problems, a band of Union Cavalry kept attacking the Confederate rear like a wolf after its prey.

General Jubal Early's Division of Ewell's Corps arose at 2:00 a.m. on the 5th and began their march toward Virginia. Their main objective was to act as the rear guard for General Ewell's Second Corps, followed by General John Gordon's Infantry Brigade and Colonel E. V. White's Cavalry. Upon entering Fairfield, General Early found a traffic jam caused by too many wagons. General Early who was not a very patient man and threatened to use blank ammunition in an artillery piece in order to sort out the wagon mess and get the teams of horses underway.

While General Early was attending the wagon situation, a dispatch from Colonel White arrived stating Union soldiers was coming. Wilbur Davis of the Charlottesville Artillery remembered the incident. He later wrote that General Early had ordered Colonel Pendleton to place a blank charge in one of his cannon and fire it over the wagon train. Just about that time Union Cavalrymen arrived near Early's Division. A soldier warned General Early about the movements and General Early turned in his saddle and looked toward the hill and saw nothing. Then a puff of smoke was seen and a cannon shell landed near the Charlottesville Artillery. It was followed by a few more shots, but did no damage.

As the Union troops approached General Gordon's flank, they were met with artillery and small arms fire. General Ewell later wrote in his report about this brief skirmish. He stated that the enemy had been threatening the rear of his Corps and they had been occasionally attacked by Union Artillery. The Federal unit eventually deployed a line of skirmishers. The Union soldiers then retreated as they were out manned. General Early reported during this small affair, the Twenty-sixth Georgia regiment sustained a loss of 11 wounded and missing.

Because of the situation with the wagons near Fairfield, Early's Division was forced to encamp that night by order of General Ewell. Early's Division was ordered to protect the trains, which was parked a little farther west of Fairfield. After repulsing Federal troops during the evening, General Gordon's men spent the night at Fairfield.

The Amherst Artillery was forced to spend the night of the 4th in the middle of the Fairfield Road in the pouring rain. By evening they encamped near Fairfield making a 10 mile march from Gettysburg as the rain and mud slowed traveling almost to a halt. By the evening of the 6th, the starving men made their camp on a farm near Waterloo where it obtained permission to feed on the farmer's livestock. During the morning of the 7th, they passed the wreckage of their supply wagons near Waterloo.

General Ewell's Corps marched into the mountain on Maria Furnace Road following Hill's Corps. When Ewell's Corps cleared Fairfield, they left behind severely wounded soldiers who were too critical to be placed in Imboden's wagon train that had already moved out of Cashtown. The rain and the dampness added to the misery. The soldiers marched through water and mud that was knee to ankle deep.

General Ewell recalled "We encamped for the night on a hill 1½ miles west of Fairfield, and next day, July 6, the Third Corps moving by another road, we were still in the rear, Rodes' division acting as rearguard, and repelling another attack of the enemy." General Ewell then continues "Attacked the troops making the summons, and drove them out of a wood in which they were posted. The enemy did not follow much beyond Fairfield. The road was again blocked till noon. That night we encamped near Waynesboro, and reached Hagerstown about noon of July 7."

At dawn, General Early moved to the front of the Ewell's Corps passing Monterey Springs where his Division crossed over to Waynesboro and encamped for the night. By early next morning on the 7th, Early's Division then moved on toward Hagerstown, by way of Leitersburg.

General Meade wrote to General Couch with concerns of the Confederate Retreat. General Meade needed reliable intelligence of the Confederate Armies movements. A captured rebel cavalry officer stated General Longstreet was moving through Jack's Mountain, and ordered him to picket roads to Emmitsburg.

Instead of the Majority of the Union Army moving behind to catch up to the Confederate Army, General Meade followed parallel on the Eastern side of the mountains in attempt to cut the Confederate Army off near Hagerstown or Williamsport, Maryland. General Oliver O. Howard was encamped at the Horner's Farm near Gettysburg. His two Corps, the Fifth and the Eleventh was getting ready to pursue the Confederate Army when he was ordered to stand down by General Meade. He wrote to General Meade with concerns that the Confederate Army might pass through Jack's Mountain to Mechanicstown and then onward to Frederick, or that the Confederate Army would travel toward Hagerstown. Because of this, General Howard wanted to move his Corps to Emmitsburg as quickly as possible to prevent any break through.

By 8:30 A.M. on July 6th, General Meade ordered General Howard to move one of his Corps to Emmitsburg and the other Corps to be posted on a road leading to Fairfield. According to General Meade early on July 6th, after receiving information on the Confederate Army's retreat route, all evidence showed that the principal force was between Fairfield and Hagerstown moving toward the Potomac River.

By 9 a.m. the Confederate Infantry numbering about 80,000 men was reported to have passed the Fairfield Road. General Meade learned the Waynesboro road was empty when parts of his army arrived. General Meade advised his Corps Commanders that he would continue his flanking movement once the main Confederate Army had retired from the mountain. With this plan he also directed General Couch to move down the Cumberland Valley to threaten the Confederate rear.

General Pleasanton ordered a brigade of Cavalry, under Colonel McIntosh, to communicate the Confederate troop’s movements as his Cavalry traveled toward Waynesboro. General George Sykes commanding the Fifth Corps wrote to General Howard during the evening, explaining his position. He was located near the junction of the Emmitsburg pike and the Fairfield road. He had not heard word from General Sedgwick on troop movements and had not received orders from General Meade or from his Wing Commander, General Howard. A sign of frustration along with the lack of communication was taking it's toll on the Union Army.

The Union Army was slow moving and several of it's officers thought that they had passed up the opportunity to end this war by destroying what was left of General Lee's Army. The unanswered question still remains. Could General Meade have destroyed what remained of General Lee's Army?

General Meade gave his report on the retreat from Gettysburg on October 1, 1863. He stated:

"On the morning of the 5th, it was ascertained the enemy was in full retreat by the Fairfield and Cashtown roads. The Sixth Corps was immediately sent in pursuit on the Fairfield road, and the cavalry on the Cashtown road and by the Emmitsburg and Monterey Passes. July 5 and 6 were employed in succoring the wounded and burying the dead. Major General Sedgwick, commanding the Sixth Corps, having pushed the pursuit of the enemy as far as the Fairfield Pass, in the mountains, and reporting that the pass was a very strong one, in which a small force of the enemy could hold in check and delay for a considerable time any pursuing force, I determined to follow the enemy by a flank movement, and, accordingly, leaving McIntosh's brigade of cavalry and Neill's brigade of infantry to continue harassing the enemy, put the army in motion for Middletown, Md."

An article printed in the Emmitsburg Chronicle tells about the Monterey Area sixteen years after the battle. On September 9, 1879, the column was written simply "One of the party":

"Dear Chronicle, a party of us left here about seven o'clock one lovely morning in August, bound for Pennmar. Being in no hurry, we had ample time to view the beauties of nature as we moved along. The first object the true attention was our own beautiful mountain, whose corpse shaped top cannot but attract even an indifferent observer, the corpse with this sheet thrown over, so perfect, that the hands folded across the breast or perceptible to all. While looking around we found ourselves in Pennsylvania, and before long had glimpses of art much talk of Friends Creek, but soon or enhancement was disturbed by the halt at the toll gate, were a little barefoot urchin took our pay. Pushing onward we remarked the Creek on our right which was flowing rapidly under the bridge on the road to Fairfield.

A great variety of ferns decked the roadside, but we could not stop to pluck them. Fountain Dale was soon passed, not far-off we saw a lake, clear as crystal which was very suggestive of rowing. Soon we passed a house without any steps in the front, the door was at least 8 ft. from the ground, we know it was inhabited, for eight or 10 children were playing on the grounds, we waive door kerchiefs to them, but they seem to diffident to make reply.

Next appear to Clairmont, and winding around the hill we were soon at Monterey, here we stopped whilst the wheels were prepared to go down the Mountain as we rode along some places seem to say "We hide goblins" so wild and dreary they looked, but being a gay party we were not timorous. We soon passed Buena Vista and shortly afterwards found ourselves in Frogtown. From the number of mud puddles, I think they must have mirth provoking concerts every evening. Looking to the left, high up on the mountainside, the observatory on High Rock came into view it seemed impossible we could ever get there, we rode on however, up, up, up, until at last we came to Pennmar Park.

It was a happy thought to select the place for a summer resort, for a lovelier spot could scarcely be found anywhere. We do not stop long as we were anxious to get too High Rock, before the excursion came from Baltimore. The site from the observatory is to grant for description, the trees below look like little bushes, so high were we above them, and there seemed no end to the country, so far was it spread out before us, one of our number saw Hagerstown, and we all saw clearly Waynesboro, Smithsburg and a number of other towns around. We left reluctantly, compelled by hunger to do so, and were soon on our way back to Pennmar.

Much to the discomfort we met carriages bringing people up from the train, the road was so narrow we had to stop to let them pass fearing they would fall over the side of the precipice, we got down safely however and were glad to rest upon the sofas was so providently in hand. The park is elegantly fitted up with a very large dining room, a dancing pavilion, dressing rooms for ladies and gentlemen, and the photographer has his place then there are swings croquet &c. Dinner was hurriedly disposed of whilst a band of music discourse sweet melody.

We left the park about four o'clock highly gratified by our long ride up there. We call it at Buena Vista Springs as we came home, and were pleased to observe how nicely and beautifully it was kept. As we passed a house without steps, the children were sitting quietly on the ground, we spoke to them where upon they all arose and serenaded us with tin pans and imitated the beating of drums with sticks on boards, and thus had bravely gotten over the rustic timidity they displayed in the morning. So ended the day full of joy to delighted with the wish that it might soon be renewed."

This group of tourists taking a buggy ride took the same route General Kilpatrick traveled to attack the Confederate Wagon Train. The scars left behind from the battle quickly heeled. The area itself became a tourist attraction as early as the late 1800's. Many spring houses shaped like mansions were built on the battlefield as the area prospered. The landscape of the Monterey area changed forever when the railroad came through. Many roads lost their appearance as copper mines dotted the area creating the main roads that we see today. Intersections changed dramatically when Old Waynesboro Pike was replaced with modern day Route 16.