ShotPut Pro Down Under. The life and times of an offloading app.

My name is Gavin Rawlings. I am the co-founder, director of photography and editor at Venture Films in Australia. I entered the industry professionally in 2008 after attending film school and I have been using ShotPut Pro since 2014.

When I was younger I always loved going to the cinemas and watching films. I always found myself wondering how they filmed movies. I loved watching the behind the scenes of my favourite films. I guess that always stuck with me. After high school I bought myself a video camera and started filming skateboarding with my mates. During this time I became more interested in camera work which lead me to film school, then eventually into the industry.

I started flying drones in 2015. I’d had a play with them before that but started

IMG_7141
The Flinders Ranges, South Australia

using them professionally then. We work on a lot of outdoor adventure shows and the landscape over here in Australia really lends itself to aerial filming. The Kimberley is always stunning from the air. The Flinders Ranges is another great spot, Cape York has some stunning locations as well. The Gulf in the Northern Territory, the Victorian High Country – I could go on and on! We were always hiring choppers and spending big chunks of the budget on them so when drones became capable of capturing good quality video it was a no brainer to start using them on shoots. I’ve always flown the DJI Phantom range of drones, currently I am flying the DJI Phantom 4 Pro+.

APES3018
The Twelve Apostles, Victoria

When I am working there are a few “can’t live without” items. My Sony F55 (I love that camera), Miller Compass tripod, Kessler Slider, CreamSource lights and ShotPut Pro. There’s plenty more gear in the kit but those would be my favorites.

Rawlings_Fotor
Cape York, Queensland

I first heard of ShotPut Pro years ago in forums. I heard people talking about this great piece of software that verifies your copied footage and I thought to myself I need to be using this. We travel to a lot of special places and only have one chance of capturing footage there a lot of the time. I want to be confident that the footage is properly offloaded and error free!

 

I think that’s my favorite feature of ShotPut Pro, the verification. That’s what helps me sleep at night! Knowing that all of the days work is safely backed up. Copying to multiple drives at once is another awesome one and I am a fan of the ‘Source Integrity Verification’ option which can help detect things like damaged card readers or cables.

I think the biggest tip I can give to beginners is to just keep filming. It’s a very competitive industry and cameras and editing equipment have become a lot more accessible. You really need to stand out. Film videos and get them up online. Share them around. They don’t have to be paid gigs, just anything that can showcase your talents. That’s what I did. I wrote letters and emails to lots of different production companies when I started out and didn’t get a response in most cases. Experiment, watch lots of films and documentaries. If you stick to it someone will notice you.

2017 Showreel – https://vimeo.com/251448192

DJI Ronin 2 Showreel – https://vimeo.com/285764799

Drone Showreel – https://vimeo.com/288290085

IMG_6120
Cradle Mountain, Tasmania

Connecting Acquisition Workflows

Screen Shot 2018-10-04 at 10.23.11 AM
Dan presented at IBC 2018 in the IABM Future Zone

Imagine Products opened its doors in 1991 and in those 27 years there has been a sea of change in the technology and methodology for video acquisition and distribution.

For example, back in 1991 the term ‘multi-media’ had just been coined and there were great debates on what that meant and how emerging digital manipulation of video would affect business and revenue models.

Being forward thinking, our company was one of the first to have a web presence and to utilize email as a communication tool, even though few others had either. Back then no one would have relied on those for commerce transactions.

Now here we are today and no one would consider transacting commerce without those tools. Likewise, I believe we are at the cusp of another new era in how visual media is handled and monetized.

Artificial intelligence and cloud technologies offer new opportunities to interact and organize, from pre-production through delivery and archive.

So what do we mean by Workflow? Simply put, workflow is the methodology you choose to accomplish certain tasks. But in our industry today, we tend to think of the term ‘workflow’ as synonymous with the process of handling digital files.

Often when performing tasks there are many ways to get to the desired result. Some of course are more efficient than others and available technology and resources limit your choices. To demonstrate how workflow changes with technology and its effect upon user experience, consider for a moment the analogy of operating a vehicle from one point to another.

In the beginning, automobiles had only manual transmissions and only one interface to them – a stick shift on the floorboard. Similarly when file based video cameras emerged over a decade ago, there were few experienced operators nor recommended ‘best practices’ in how to care for those files. It was a very manual process of dragging and dropping files, and very susceptible to human error and data loss. As time went on, there were some variations on the manual transmission such as a column attached shift lever, but the human interaction only changed slightly.

When automatic transmissions appeared in automobiles they made the entry-level operation much easier and took a lot of the guesswork out of interaction. The workflow was dramatically improved through a technological advancement.

Likewise in video file handling, new tools emerged. Specialized copy utilities helped ‘data wranglers’ organize and reduce potential human errors, and alerts for equipment failures made it easier to get good results, even for novice users.

Now, much like the advent of self-driving automobiles the emergence of artificial technology as applied to imagery is cutting edge. While the advantages are clear, it’s just beginning to become a practical option. And like all new technologies, how we apply AI and cloud tools will depend upon cost justified use cases and efficiencies to be gained. So are we ready for a fully autonomous video workflow from acquisition through archive?

Probably not. But the opportunity for some intermediate steps in the near future are coming into focus.  For example, many are already using the cloud for repositories and distribution options. But few have connected the ingest phase in a meaningful way or used AI for large scale indexing of metadata tags during acquisition.

Once indices are created and cloud shared, others will be able to access them for grouping, filtering and searching. Coalescing information and assets from multiple ingests can be a daunting task for large, multi-camera shoots, and is a perfect application for more automated AI and cloud interaction utilities.

Such tags could carry over from original material through to finished archived projects, and later paired with full resolution files should they too be uploaded or referenced in the cloud account.

Since AI involves mathematical analysis of video frames to interpret clues about its contents, it should likewise be useful in immediately detecting corrupt files and alerting the user to potential equipment failure. The metadata of bad files could be linked back to specific data sets coming from the same equipment that may be suspect. The overall consistency of video frames could be analyzed in purposeful ways to enhance quality control.

When we consider the data wrangling workflows of today, much still depends upon human tasks. Most of the popular applications and equipment as yet do not seamlessly pass information, relying instead upon operators’ diligence and organization skills for best results.  Scene detection, voice, optical character and facial recognition technologies have cumulated in artificial intelligence services for images and video clips. But AI for imagery is in its infancy and still resource greedy. Look for audience appropriate indexing and filtering to assist in meaningful ways with the creative process.

In the short term the costs will force as much processing as possible to the local computer or equipment. For example, you’ll see mostly pre-processing of video clips before uploading to the cloud services to reduce bandwidth, storage and processing costs. Eventually we may see the availability of AI toolkits for use on desktops or incorporated into hardware accelerators.

What workflow you employ can have a dramatic effect on efficiency and profitability. Like manual and automatic transmissions, to self-driving automobiles, there are many ways to accomplish the same task. And more people than ever are involved in the moving of media files. Which workflow lane will you be in?

Happy Anniversary, ShotPut Pro! The life and times of an offloading app

It’s the 10th anniversary of ShotPut Pro! We’re celebrating 10 years of making life easier for anyone who’s responsible for digital media files – and reflecting on the past.

At the time we invented ShotPut Pro, affordable HD video was just coming into being. Prior to that, everything was tape-based. Expensive cameras were recording to HD tape that had to be played on expensive players.

Panasonic and Sony were the leading camera manufacturers back then, and they were 4gbp2_LRGdeveloping direct-to-digital-file cameras. Panasonic had just introduced its first P2 camera under $10,000 to bring HD to the masses. It held two P2 cards at about 4 gigabytes each. When you filled one up, the camera automatically jumped to the second card and started filling it. At that point, you either popped out the first card and put in a new one, or you quickly got the material off that card, cleaned the card, and put it back in the camera. Manufacturers and users were only just beginning to figure out the workflow.

That’s when Imagine Products developed an instantaneous player for the P2 media called P2 Log. P2 Log was a breakthrough because, until then, there was no way to see the video you took out of the camera until you transcoded it to QuickTime, which took real time. Imagine if you had 10 minutes of material, and it took 10 minutes to transcode before you could watch it on something other than the camera.We made a player that could do that instantaneously and in doing so, we became familiar with the technology and the file formats.

So when Sony decided to do something similar with SxS cards and XDCAM EX – the competitor to the low-end Panasonic camera – Sony commissioned a similar tool for viewing the footage. In addition, Sony wanted a way not only to move that material quickly from the cards onto a hard disc, but to verify it was copied correctly before erasing the card. Imagine Products’ answer was ShotPut-Ex, an offloading tool made especially for Sony.

When the Panasonic folks saw what we developed for Sony, they wanted it too, so we made a ShotPut-P2 version of the software. Then the RED camera people saw it, and we developed a version for them. Now we had three versions of ShotPut. Finally, when ARRI came calling, we decided to stop making separate versions for each camera type. Instead, we created a tool that could detect the camera type and work with all the various file-based parameters and work for all of them.

And so, ShotPut Pro was born. 

Before ShotPut Pro came along, people were simply dragging and dropping files from Imagine_ShotPutProone place to another without verifying anything, assuming everything was OK. Many people didn’t realize the importance of copying every single bit of data that was on that card, such as the audio files or the metadata. They often discovered – too late – that there was a problem downstream, when it came time to put the material into an editor. With the old tape-based workflows, the original material still resided on a physical tape, so you could always go back and redigitize it into the editor if something happened to the digital file. Not so with files on cards. Once you’ve done the drag and drop and cleared the card, the original is gone. There’s no going back.

So we began to make other tools for things like repairing damaged file structures. The file-repair function helped people recreate the missing files and check that all was well.

For the first few years, cards still had to be returned to the camera quickly because the storage size was so small and the cards were so expensive. (A 4-gigabyte P2 card was in the neighborhood of $1,200!) Thankfully, over time, cards got bigger and cheaper. But it always came back to verification: “How do I know things got copied properly?”

As people became more comfortable with digital-file workflows, they needed more features. Indices didn’t exist at the time, so people wanted a printout showing what footage existed within a session or project, instead of having to load each hard disc to find out. Enter ShotPut Pro’s reporting function, which helped keep projects organized.

A few years later, the DIT position became a thing, and professional data wranglers emerged. Then, it wasn’t long before insurance companies got involved, demanding checksum reports and an audit trail showing which camera the files came from and where they went, and proving that the copies were identical to the original. As a result, demand for ShotPut Pro grew.

And so, ShotPut Pro matured as the industry evolved. Every time an operating system changed, ShotPut Pro changed. Usually, the changes took place in the background and had to do with file handling. For example, ShotPut Pro never just did a simple copy to Finder or Explorer. Instead, there were always low-level copy functions going on behind the scenes, such as byte comparisons to verify that original and copied files were exactly the same.

Over time we added even more features and functions that simplifies proper digital offloading.

Presets were a major improvement because they made it possible to store commonly used naming conventions, output destinations, and other parameters rather than rebuilding them every time. Later versions came with other options, such as automated file naming. Presets became more and more important as more professionals got involved and the volume of content increased. They make it easier for numerous people working on the same project to track things and stay organized.

Back in the early days, ShotPut Pro would simply copy a card onto a hard disc and create a folder with whatever name you gave it. All the contents of the card went into that one folder. But now things are more complicated, and you want to have a folder and subfolders for the project, the camera, the shoot dates, and the specific cards.

Metadata sensing also became important, it’s a routine that examines the video; determines the type; extracts information such as frame rate, duration and timecodes; and extracts thumbnails. As a result, you can run reports containing one or more thumbnail samples, so you have an idea of what each clip was about. Metadata sensing was a massive change and a major improvement, especially because a lot of producers use those PDF reports to communicate with their clients.

All along, we wanted to make a product that would yield good results without a huge learning curve. The beauty of ShotPut Pro – and why people like it so much – is that it’s very simple for a beginner to use, yet it has many layers of complexity. The deeper you dig into the features and options, the more sophisticated it becomes, so it appeals to everyone from super pros to casual users.

When we first launched ShotPut Pro, a casual observer might have seen it as “just a copy utility”. And it was! But more importantly, it was – and still is – all about making things easier, trying to eliminate human error. Because let’s face it: A lot of time you’re tired. It’s the end of the day, and you just want to get done and go to bed. That’s when mistakes get made, and that’s when ShotPut Pro tries to keep people from doing foolish things.. like erasing a card before it has been copied or overwriting the original.

Thanks to all these cool things, ShotPut Pro has had a major impact on the industry and has earned its place as the way to offload and verify files. Because we entered the digital-conversion revolution early, ShotPut Pro became the standard. There are many ways to copy files, obviously. The questions are, how secure are they, how easy are they, and how fast are they? ShotPut Pro is quite frequently specified by insurance companies, TV and movie producers, dealers because it is all three, and they trust it implicitly.

So where will ShotPut Pro go in the next 10 years? Stay tuned!

LTO Best Practices Part 3: LTO 8 and Beyond

Before reading this blog post, please consider reading Parts 1 and 2:

Part 1: You’re Settled on LTO/LTFS, Now what? 

Part 2: Advanced LTFS Software Can be the Key to Success.

It’s time to get back into the LTO discussions! LTO 8 has been shipping for a few months now, here’s the skinny:

LTO 8 graphic-01

So basically, LTO is here to stay and its becoming more affordable with denser capacity! That’s awesome, right? Right! In this blog series we’ve defined LTO/LTFS archiving and why it’s a secure form of long-term storage. We’ve shared best practices through out all parts and we will do that here again. But we’d also like to share what other industries are turning to LTO/LTFS use and why, as well as diving a little deeper into the use in the M&E industry.

We’ve been pleasantly surprised in the scope of our customers here at Imagine Products. We’ve been working in the media and entertainment industry for years (27 but who’s counting!) and the longer LTO/LTFS archiving is out there the more diverse our customer base becomes.

We have many real estate offices that use PreRoll Post to archive files. And this make sense because they obviously have a TON of stuff to archive – have you ever bought a house?? The amount of paperwork is daunting… now think about storing all of that. LTO/LTFS makes perfect sense. We have several churches who use myLTO to create archives as well. Which is great because it’s a low cost, long-term storage solution for them. It’s also simple to use, so almost anyone can be trained on it. There are also doctors offices and law firms that are now archiving with LTO/LTFS hardware and software.

At NAB this year we learned that many of our partners in the archiving realm have similar customers. The LTO consortium has done a great job of making this open source easy for everyone to adopt.

LTO.org published a white paper a few days ago which does a great job outlining how LTO/LTFS archiving is being used in the media and entertainment industry. It shows how we’ve moved past just archiving finished projects and are now utilizing it throughout the entire process.

Screen Shot 2018-04-23 at 2.42.45 PM

This is great because it tells us that LTO/LTFS is widely accepted, and here to stay. As if the peace of mind from knowing that the LTO consortium was backing this wasn’t enough!

The white paper, which can be read in its entirety here, details how Sim Digital uses LTO/LTFS to manage workflows. Here’s that excerpt:

For SIM Group, a post-production technology and services provider, the challenge was to come up with a storage solution that could be used to easily by local production teams and that would also protect the group’s media assets in a secure manner.

SIM – which specializes in on-set data management, digital dailies editorial system rentals, and data archiving – could not always afford to send their own production team out on production shoots. So they often looked to local production resources to handle camera media offload and backup dailies. But that meant they needed a solution that was simple to use and required minimal training.

SIM Group also wanted a flexible solution that allowed them to support widely varying workloads; sometimes they would have many projects, and at other times just a few. They were looking to maximize their production efficiencies while safeguarding customers against unanticipated costs. The firm wanted to protect each day’s production to reduce the need for any reshoots.

To accomplish these goals, SIM Group implemented a workstation application from Imagine Products to transfer camera media to LTO with LTFS tape. The solution allowed SIM Group to index camera originals automatically and scale their usage of the system as their workloads fluctuated. A streamlined setup such as this requires only a workstation and an LTO tape drive, which makes it simple enough for non-IT staff to operate the system smoothly.

“Before LTFS came along, a lot of time and money was put into figuring out how to de-archive other companies’ tapes,” says Jesse Korosi, director of workflow services at SIM Group. “With LTFS tapes being recognized in so many different software options, post companies like our are saving money not having to build out a new workflow/system for every job we receive that needs to be de-archived.”

Side note: Congratulations to Jesse Korosi for being honored at the NAB show this year with a Top 50 Honoree award from Studio Daily!

So where do we go from here? 

LTO/LTFS is here to stay, that’s pretty clear and if it wasn’t – the consortium has published a road map for the future with LTO 10 having a capacity of 60 terabytes! Looks like storage will continue to get more secure, more affordable and more widely used. We aren’t going anywhere! #archivesecurely

 

Thunderbolt 3 – It’s FAST!

Speed, speed, speed. We talk about it all the time in our blogs because it’s one of the top concerns for our customers. Everyone is looking for the fastest technology to get the job done. Today’s blog is no exception! Let’s talk about Thunderbolt 3.

At a max speed of 40 Gbps it’s no real shock that this is the preferred connectivity for many in the M&E industry. Thunderbolt 3 holds so many advantages over its predecessors. Avram Piltch said it best, “If you want to connect your notebook to multiple 4K displays, attach a graphics amp, transfer giant files to the fastest external drives or grab RAW video from an expensive camera, you should get Thunderbolt 3.”

He also wrote a great article on Thunderbolt 3, which you can read here.

It is he discusses some huge benefits of Thunderbolt 3, including:

  • 4x the Speed of the Fastest USB Connection
  • Uses USB Type-C Connectors
  • High-Speed, Peer-to-Peer Networking
  • Daisy Chain Up to 6 Devices

Thunderbolt came from the brilliant minds at Intel and they’ve clearly been perfecting it ever since. Right out of the gate, Thunderbolt was blowing USB data transferring rates away at a two to one ratio. Thunderbolt 3 has taken speed to a whole new level, doubling the transfer rate of Thunderbolt 2 from 20 Gbps to 40 Gbps. That’s more than enough to work with 4K video!

So not only is it super fast, it also works beautifully with USB-C connections – that’s because it’s the same size port! Thunderbolt 3 cables with work with USB-C cables and USB-C cables will work with Thunderbolt 3 cables. ONE CABLE TO RULE THEM ALL!

ring
You KNOW I can’t resist a good movie reference!

The only downside is if you want to use Thunderbolt 1 or 2 ports, you’ll need an adapter and obviously that will affect the speed. As we get further and further into 2018 more and more Thunderbolt 3 devices are becoming available.

Check back for our next blog post!

blog bio-01

Imagen: 10 Things You Need to Know About Cyber Security

I recently read (most of) a white paper put out from Imagen about Cyber Security (- it was 36 pages, give me a break!) There was a lot of great information in, which I will paraphrase below. If you care to read the white paper, you can find it here!

  1. Know Your Enemy – there are 5 potential threats both in and outside of your organization.
    • Casual Hacker – someone who is curious about your system and may cause minimal disruptions.
    • Hacktivist – an activist campaigner has become more and more common lately with the political turmoil around the world. These generally have an agenda and are centered around a perceived injustice.
    • Organised Crime – these are groups whose sole purpose is to steal information and sell it.
    • Nation states – remember the Sony hack? Many believe that this attack was made by the country of North Korea in retaliation for the movie The Interview.
    • Internal threats – past or present employees have access to company’s information and can take advantage of that access.

There are a number of ways to prepare and prevent these kind of attacks. Prepare and plan; keep websites, software and anti-malware programs up-to-date; educate your team on how to recognise scams; use secure Internet connections whenever possible.

2. Be an informed decision-maker – be sure that you or someone in the organization is responsible for preventing and preparing for a potential cyber attack. This should be someone with not only knowledge but the ability to make financial decisions to protect the company.

3. Use security tools – in this industry we love tools and toys so this one completely makes sense! Use up-to-date technology to protect your assets, even down to the rushes you create. Be sure that connections are secure while information is being passed back and forth and that content such as scripts, calls sheets and contact details are all protected as well.

4. Secure your smartphone – we can do almost everything from our smartphones including shooting an entire movie! This is the most common unsecure place that information can easily get stolen. Using a longer passcode, installing anti-malware, enable whole disk encryption and find my phone are just a few options to help keep it safe.

5. Beware of free WiFi – this easily opens up your phone and computer to hackers who are more than happy to steal your information – and it’s quite easy to do. This can even occur from a hotel wifi so ask questions and know what wifi service you are connecting to. Stay off of banking sites and social media while connected to free wifi, these make stealing your identity much easier. If possible pay for a VPN service.

6. Understand the content lifecycle – live events have a higher value when they are being aired or right after but quickly lose their value. A primetime drama production has different value and risk at different parts of the process. It’s important to consider the project and the risks at different stages and act accordingly.

7. Know the law of the land – we all travel – a lot. It’s important to remember that laws change and security levels change depending where we are. Removing sensitive material from electronic devices before traveling is probably the easiest and also one of the best safeguards. It may also be a good idea to encrypt things if they are being exchanged from one country to another.

8. Know your friends – a good hacker will know how to get a username or password out of a potential victim and responsible organizations will never contact you and ask for personal information. Spam filters are set up on email services for a reason – use them!

9. Dispose responsibly – when you return rental equipment, make sure you wipe all personal information from it. There are free tools that can be downloaded that will securely erase your data, remember pressing delete doesn’t always completely remove something from the computer.

10. Be prepared by planning – it you expect it and plan for it, then you can minimize the damage when it happens.

There’s a lot of great information and examples in Imagen’s guide. This brief synopsis was meant to make you think and consider your security. For a full account and detailed suggestions, read the full version here!

 

 

 

 

Apple Security Issues… What’s up with that?

By now all Apple users have heard and (hopefully) upgraded to 10.13.1 to avoid the security issues present in 10.13. So what heck actually happened and what did Apple have to say. Let’s take a look:

Many of us got a notification on the morning of Wednesday November 29th that we needed to do an update immediately that related to security on our computer. I’ve been a Mac user for a long time now (10+ years) and I don’t remember seeing an update like that before! Like a good Apple user, I complied… immediately. And like any Apple user, I took to Google (yeah, not Safari 😁) to figure out exactly what was going on!

The discovery was posted to Twitter (my inner customer service rep shutters at the thought of this kind of bug being reported via social media) and the frenzy began!

Screen Shot 2017-12-04 at 2.49.59 PM

So basically that meant that anyone can log into a Mac by entering “root” as the username without a password. The first time you try to login, it won’t work. But if you try it again, you will be granted access.

According to Apple, “The user account named ”root” is a superuser with read and write privileges to more areas of the system, including files in other macOS user accounts. The root user is disabled by default. If you can log in to your Mac with an administrator account, you can enable the root user, then log in as the root user to complete your task.”

So as you can see allowing just anyone access to a “superuser” account is kinda bad!

To Apple’s credit the response was swift. Within hours they had a fix and within 24 hours they were pushing it to all their users. And I’ll give them credit… they messed up BUT their response was perfect. They corrected it and owned up to the mistake.

Screen Shot 2017-12-04 at 3.37.42 PM

Here’s the support notes for the security flaw.

Maybe now they can work on fixing iOS 11… 😳

blog bio-01

New Filmmakers. New Funding model.

“Three years after raising its curtain on operations, Bloomington-based Pigasus Pictures is making its mark, with one award-winning feature film to its credit and another with growing expectations set to wrap up production within a month.

And the firm’s two young Hoosier founders say they’re ready to accelerate production even faster.

The company’s early success has put a spotlight on a unique financing method founders Zachary Spicer and John Armstrong are using for Pigasus’ projects, including three more feature films and two television pilots. All that work is slated for production in 2018 and 2019, with initial release by 2021.”

I stumbled upon a great article from the IBJ this morning and wanted to share it not only because it comes straight out of Indiana but it also outlines a unique outlook on feature films, budgets and quick timelines for production. The production company, Pigasus is “a small-town production company with a forward-thinking, community-focused approach to filmmaking.” Rooted in Bloomington Indiana, owners Zachary Spicer and John Armstrong say they shoot all over the state and edit in Bloomington.

‘“Indiana has a lot to offer in terms of landscapes,” Armstrong said. “From the flatlands of northern Indiana to the hills of southern Indiana, it’s very diverse.”’

Their unique funding plan is what really caught my eye!

“The Pigasus founders aim to raise $6.5 million to produce “Ms. White Light” and the company’s other five upcoming projects. The duo said they’re off to a good start and feel confident about reaching their goal but declined to say how much they’ve raised so far. They have set up a private equity fund as a separate limited liability company. Pigasus is manager of the fund—and has a small stake in it. There is no limit to the number of investors, Armstrong said. Pigasus can allocate some of its ownership share for each production as an incentive to actors or any creative partner. Investors get paid back first, Armstrong said. Once investors “are made whole,” he explained, the creative team and actors get a share of the profit based on pre-arranged agreements.”

Read the full article here to learn more about their funding and distribution process.

The Good Catholic Trailer – Pigasus Productions

 

And because they’re Indiana University alums (and it’s my alma mater), I will leave you with this gem:

 

blog bio-01

Checksums and Verification Part 3: Speed vs Security

THIS IS A MULTI PART BLOG. AS MORE BLOGS ARE POSTED, LINKS TO THOSE POSTS WILL BE INCLUDED IN THIS BLOG.
BEFORE READING FURTHER, WE ENCOURAGE YOU TO BEGIN WITH: 
CHECKSUMS AND VERIFICATION PART 1: THE 5 W’S WHICH DEFINES WHAT A CHECKSUM IS AND WHY IT’S IMPORTANT IN THE MEDIA AND ENTERTAINMENT INDUSTRY.
CHECKSUMS AND VERIFICATION PART 2: DEFINE AND DECIDE WHICH DEFINES VARIOUS CHECKSUMS AND WALKS THROUGH HOW TO TEST A CHECKSUMS VALIDITY.

Speed: Let’s face it, we are all constantly trying to do everything faster. When my computer takes two minutes to boot up, it feels like 2 years. When a page doesn’t load IMMEDIATELY it seems like the world is ending, slowly and painfully <forget about the fact that I grew up during the era of dial up, in fact we didn’t even have a computer in the house until I was 14! Now my family of 5 has…. 5 computers!> Nope, time and speed are always something on our mind. And in the media and entertainment industry it’s paramount – see what I did there 😉

So when it comes to offloading securely and quickly there can be a lot of factors that can seriously affect it. Here are some friendly reminders and things to consider when offloading, especially when you are considering foregoing security for speed.

Bus speed and drive speed:

speed
the bus jump” from the 1994 action thriller Speed

Keanu taught me that the bus speed has to stay about 50 mph and everyone knows time travel requires 88 mph… wait, wrong kind of bus, wrong kind of speed.

Ok what’s a bus and what’s the speed have to do with it: The bus is a data connection between two or more devices connected to the computer. A bus enables different parts of the computer to communicate with one another. A computer or device’s bus throughput is

The connection between the hard drive and laptop is connected via bus and is therefore affected by that bus and it’s throughput. And as Jason Johnson so keenly put it, “This is why a RedMag on USB 3 will only offload at 120MB/s. The card’s bus for data transfer is the limiting factor. The same goes for Lacie Rugged drives, etc.” (If you don’t know Jason you should, he’s a wealth of information!) Although USB 3 has a throughput of 640 MBps, the transfer is limited to the throughput of the RedMag.

Vintage-HEINEKEN-Stretched-Long-Neck-Glass-Beer-Bottle-_57
Wrong kind of bottleneck but it’s cool and we just got back from IBC in Amsterdam, home of Heineken!

Different connections, have different speed throughputs. This is important to remember! Whatever part of your workflow has the slowest speed or throughput is your bottleneck.

Ryan Nguyen also blessed me with this wonderful metaphor: “USB 3 and Thunderbolt 2 are like speed limits on the highway versus the autobahn. Thunderbolt 2 certainly allows you to travel faster, but your real world speed depends on multiple factors including the weather, traffic conditions, car tire performance, etc.” I love this example because it reminds us that there are many factors to consider when offloading or copying files or folders. It also takes me to my next point and again I will use Jason’s words, well, because they are spot on!

“Copying one large file is faster than copying multiple files. ProRes versus DNG or ARRI comes down to a series of handshakes. Imagine making a deal and shaking hands. That is ProRes. Now imagine making hundreds of smaller deals and shaking hands each time. That is ARRIRAW. Each side needs to stop and start the action every time a file is transferred.” Yet another factor, and sometimes you can’t avoid transferring hundreds of files – that’s ok! It’s important to realize that you have to adjust your speed expectations.

Security: We’ve established in previous posts what checksums are, how to determine which one to use and if your content is actually being verified – if you’re behind read Parts 1 and 2 now!

Checksums are another factor to consider when thinking about speed. Copy and Paste is a bad practice (as we all know) because there is nothing ensuring the copies are 100% accurate. There’s nothing saying “heck yeah man, we got it all” which, by the way, is what every checksum in the world has ever said.

So what do you do when you need something offloaded quickly and securely OR (and this is dangerous but we’ve had the request so it must be addressed) you’re willing to forego some security for more speed?

If you need speed AND security find the right checksum: that’s currently XXHash.

xxHash – xxHash is an extremely fast, non-cryptographic hash algorithm, working at speeds close to RAM limits. It is proposed in two flavors, 32 and 64 bits. (SMHasher on github.io)

For ShotPut Pro, ShotSum and PreRoll Post we use xxHash 64 bit. We recommend using XXHash as the checksum type unless you have a requirement for some other type. XXHash can out perform MD5 for example because it can go at the speed of your RAM whereas MD5 is a CPU dependent process.

If you need less security and more speed you can opt for a file comparison verification over the checksum verification – remember you’re not going to get the “heck yeah man, we got it all” (see Wayne above) with this option. But it’s a possibility and one that has it’s time and place. In ShotPut Pro and myLTO we call it File Size Comparison and we even go a step further and give the option of just calculating the checksum for the source for later reference. Again, not best practice but there is a time and a place for that kind of verification and something to think about when speed is your number one consideration.

Offload confidently and be on the lookout for Checksums and Verification Part 4. If you have something you’d like us to discuss, please leave a comment here!!

And now, this…..

maxresdefault

You didn’t really think I would mention Back to the Future and not include and awesome shot of Doc and Marty did you??? 1.21 jiggawatts!!!!

macOS High Sierra – How does it affect our workflow applications.

Here’s the good news – ALL Imagine Products applications are High Sierra compatible. But before you hit that update button, let’s take a quick look at what High Sierra is and what you can expect.

From Apple:

New technologies at the heart of the system make your Mac more reliable, capable, and responsive – and lay the foundation for future innovations. macOS High Sierra also refines the features and apps you use every day. It’s macOS at its highest level yet.

New Features:

  • Apple File System – for a complete overview check out our APFS blog – this allows ShotPut Pro to support the file cloning feature – which saves drive space
  • 64 bit architecture for flash technology with the ability to scale for future uses
  • common tasks happen instantly, iApps have all gotten face lifts and added features
  • built-in encryption, crash-safe protections
  • more powerful GPU and the ability for apps to utilize GPU more efficiently like PrimeTransocder
  • High Efficiency Video Coding – H.265 – hooray for more 4k

So there are the quick facts! Imagine is ready for High Sierra, are you?