Review: Adobe Premiere Pro Auto Reframe
If you produce video for traditional video outlets, it’s pretty much a 16:9 and 1080p world. If you produce for social media, however, you have an array of aspect ratios and resolutions to deliver, including 9:16, 1:1, 4:5, and many others in a variety of platform-specific resolutions. While outputting your 1080p or 4K source to these requirements is straightforward, making sure the most relevant content from your widescreen video is effectively presented in those non-traditional outputs can be a time-consuming challenge.
Figure 1 illustrates this problem. This shows Premiere Pro’s Program monitor previewing a 9x16 video of the opening sequence of Dua Lipa's "New Rules" video that I downloaded from YouTube, which walks the singer through a gauntlet of rooms, hallways, and girlfriends seemingly in one long continuous shot. The video was shot in 16:9, but I’m producing it in 9:16, say for Snapchat or Instagram. It’s an extreme case, but it illustrates the problem.
In the Program monitor, the 9:16 window showing the singer is the visible component that will be output with the video. The bounding box around the visible frame is the 16x9 input source. The blue dots in the middle of the frame are the keyframes used to keep the relevant action within the output window. There are about 156 keyframes in this 62-second segment; if it took you a conservative 30 seconds per keyframe, it would take you well over an hour to position and insert them. On my HP Zbook notebook, Auto Reframe generated them all in under ten seconds.
Figure 1. Converting a 16:9 source into 9:16 output.
From a project creation perspective, this is a very simple example. In the original edit of this video, the sequence would be comprised of dozens of clip segments. If you applied Auto Reframe to the original sequence, Premiere Pro would make a similar adjustment to each segment in the sequence, and the result should be quite similar. I’ll explore operation in more complicated projects below.
At a high level, Auto Reframe lets you edit your video as normal, and when complete, output in multiple resolutions and aspect ratios via simple commands. As part of the process, Premiere Pro analyzes all components of the sequence and shifts them within the new output resolution to make sure the region of interest is always in view, creating keyframes you can adjust as necessary to perfect the video. In my tests, Auto Reframe was incredibly fast and very accurate.
Auto Reframe Operation
You can access Auto Reframe in one of two ways. First, many social media producers have sequence presets specific to the desired output resolution and aspect ratio. These are simple to create in the New Sequence panel shown in Figure 2 where you see a 600x600 preset suitable for square videos on Instagram or Pinterest.
Figure 2. Creating a 600x600 preset.
In this mode of operation, you would create and edit the sequence. Before Auto Reframe, you would then manually position your typically 1080p video within the 600x600 frame to optimize the content in the video. For even short productions, these adjustments could take quite a while to perfect.
With Auto Reframe, you drag the effect onto the clips in the sequence as normal. By default, Premiere sets the vertical resolution of all videos to the height of the preset, which is 600 pixels in this case. When applying the effect, you tell Premiere how much motion is in the video by choosing between three motion presets—default, faster motion, or slower motion—which I explore below. Premiere Pro then analyzes the clips and inserts keyframes.
Figure 3. Auto Reframe applied to the Dua Lipa clip for "New Rules."
Figure 3 shows the operation within Premiere Pro with the same Program Monitor as Figure 1 on the right. On the left you see the Effect Controls panel with keyframes shown as diamonds. These keyframes adjust the horizontal positioning of the video and follow the motion with very good accuracy in this case. That is, in reviewing the 156 keyframes Auto Reframe produced, I would have adjusted about five of them to position the content more effectively.
Applying Auto Reframe to a 1080p Production
Using a custom preset is one way to apply Auto Reframe; the other is to apply the effect to an existing sequence and specify the output aspect ratio you want Premiere Pro to produce. This is probably the workflow most producers will utilize since it lets you build the 16:9 version of your clip and create additional sequences for your social media targets from there.
Note that you apply the effect by right-clicking the clip in the Project panel, not in the timeline. Once you apply the effect, Premiere Pro will create a new sequence at that aspect ratio, appending the aspect ratio to the sequence name as you see in Figure 4. You choose from the same three motion presets described above and elect to either nest the original sequence into the new sequence or create a separate sequence. More on nesting in a moment. Premiere Pro inserts all new sequences in a separate bin in the Project panel named Auto Reframed Sequences, but you can drag them into another bin once they are created.
Figure 4. Creating a new sequence at a 4:5 aspect ratio.
Figure 4 created a 4:5 video from the red-carpet sequence at the start of the first Zoolander movie. In Figure 5, you see the 4:5 output and the keyframes created on the left. In this 30-second clip, Premiere Pro took five seconds to produce the 56 keyframes, and in this more forgiving aspect ratio, I would have only adjusted one or two of them. This bodes very well for producers converting 16:9 source footage to either square 1:1 or vertical 4:5 output.
Figure 5. Converting a 16:9 video to 4:5 output.
Nesting vs. Non-Nesting
Regarding the nesting vs. non-nesting decision referred to above, Figure 6 shows the two alternatives, which will represent a tough choice for some producers. Here’s why. Assume you’ve created a sequence that has both motion adjustments and transitions, which isn’t all that unusual. For example, you may have nudged your video a few pixels vertically or horizontally to optimize rule of thirds positioning or to eliminate an object in the background. You might also have added some fades to black between scenes or 2-3 frame crossfades to smooth out camera angle switches.
If you choose the first option and elect not to nest the clip when creating the new sequence, Auto Reframe will restore all videos in the sequence to their native resolutions, eliminating the motion adjustments but preserving the transitions. If you choose the second option and nest the clips, you keep motion adjustments but lose transitions because the clip handles at the start and end of each clip don’t transfer with the source clips.
Figure 6. Your options when creating a new sequence via Auto Reframe.
Half of this choice is illustrated in Figure 7. On the left is the source 16:9 clip. The clip has timecode burned into the bottom, which I eliminated by zooming into the clip slightly and shifting the frames downward. When I produced the square 1:1 sequence via Auto Reframe on the right, I didn’t nest, and you see that Premiere Pro reversed the motion adjustment and created the new clip using the full resolution of the source, revealing the timecode on the bottom. When I recreated the clip and nested, Premiere Pro preserved the motion adjustments, but had there been any transitions in the clip, they would have been eliminated.
Figure 7. Original clip on the left, square 1:1 clip created by Auto Reframe on the right.
Figure 7 also shows the ability of Auto Reframe to shift titles and motion graphics around to optimize presentation in the new clip. On the left, in the source clip, the title is off my left shoulder. In the square 1:1 clip, Auto Reframe shifted it inwards to make it completely visible in the new clip and made it smaller. This works because I created the title in the new Graphics panel, and Premiere Pro should similarly adjust all titles and motion graphics produced in Premiere Pro or After Effects. For all other graphic components, Auto Reframe transfers them to the new sequence without similar adjustments.
You see this in Figure 8. On the left is the 1080p original, with new titles created in the graphics panel on the left and legacy titles on the right. When I converted this to a 1:1 production via Auto Reframe, Premiere Pro adjusted the size of the circle and text in the new titles and shifted their positioning within the frame, but adjusted neither the size nor position of the legacy titles. You'd have to adjust either way, but if you’re looking for a reason to finally ditch legacy titles and use the new schema, Auto Reframe might be it.
Figure 8. Auto Reframe adjusts the size and positioning of new titles but not legacy titles.
Note that while you can choose a custom aspect ratio with Auto Reframe (see the bottom option in Figure 3) you can’t choose a custom resolution. So, if you want 400x400 output, you’ll have to choose Square 1:1 output, take the resolution Premiere Pro gives you (1080x1080 for 1080p source) and output to the desired resolution in Adobe Media Encoder.
Other Tests
We tested the three motion settings shown in Figure 9 on several clips, including the conference video shown in Figure 10, which was a panel discussion originally produced at 16:9 and converted to 1:1. In the clip, the camera shifted from speaker to speaker but was static at each speaker. You see one such close up in Figure 10. In this case, with the static camera, any motion was simply normal movement of the speaker which you wouldn’t really want to shift the frame to follow.
Figure 9. The three motion settings available in Auto Reframe.
In a 30-second segment with Slow Motion selected, Auto Reframe inserted 45 keyframes that were all tightly focused on the center of the frame and pretty much unnoticeable. With Fast Motion selected, Auto Reframe inserted 83 keyframes that moved the frames over a much greater distance, which was distracting. Similar experiments with faster moving clips showed similar results. So, it definitely pays to match the setting to the motion in the clip, though in this case completely manual adjustments might have been the best option.
Figure 10. The blue dots in the middle of the frame show the spread of keyframes in this 30 second relatively static scene.
Complicated Projects are Complicated
Most of my early testing was on simple projects, basically a single video on a timeline. Not surprisingly, the results were fabulous. Then I started applying Auto Reframe to more complicated projects. In general, so long as the projects weren’t graphic-intensive, the results were quite good, and I could have used the output almost without adjustment. For example, one project was a short concert clip consolidated from three camera angles. Though the 1080p song I converted had dozens of short segments from the camera switches in the sequence, Auto Reframe operation was quite good, and I could have used the 1:1 output with minimal adjustments.
On the other hand, graphics-intensive projects were more complicated. One project, shown in Figure 11, was a haunted-house promo where I inserted a frame around the video to create an old movie-like effect. When converting the 1080p project on the left to the 1:1 output on the right, Premiere Pro adjusted the frame so that it truncated the top and bottom.
Figure 11. Graphics complicate projects converted with Auto Reframe.
To be sure, adjusting the frame manually is a simple fix I would have had to perform to output the clip to 1:1 before Auto Reframe, and the automatic adjustments made to the videos in the sequence will save loads of time. The point is that for some projects Auto Reframe will be one-click simple, and for other projects, particularly those with graphics, it won’t be. Those working with complicated, fine-tuned projects will have to experiment with the various options described above to identify the best workflow.
Limitations
The main limitation in this initial version is that Auto Reframe only adjusts horizontal or vertical positioning and not scale. So, if you’re working with 16:9 source, Auto Reframe will only make horizontal adjustments like that shown in Figure 1. If you’re working with 9:16 source like videos shot in portrait mode on your phone, Auto Reframe will make only vertical adjustments. It never adjusts on both axes, and it never adjusts the scale.
To explain, if you had an unlimited budget to convert the Dua Lipa video to 600x600 resolution, you’d almost certainly want to adjust size and vertical positioning as well as horizontal. For some producers, Auto Reframe might be a great start, but others may prefer to work with a blank slate and set keyframes for all three adjustments as they go. All that said, for the vast majority of producers, Auto Reframe will give you a great start if not get the job done completely.
The other issue with Auto Reframe is that it’s only available in Premiere Pro version 14, which is only available for Windows 10 and not Windows 7, which I still run on my primary HP Z840 workstation. Similarly, the latest version of Premiere Pro is only available for Mac version 10.13 (High Sierra) and later. If you’re a social media producer resisting upgrading, Auto Reframe may be a deciding factor.
The author thanks Francis Crossman, Adobe Product Manager, Premiere Pro, for taking the time to discuss Auto Reframe’s goals and operation. That said, any and all errors in the article are strictly the author's.