If you attempt to start an H.264 ISO recording a see a "Recorder Error" message (shown below), it's typically related to Studio's project format and how your system's hardware encodes video.
Studio has the ability to create an H.264 ISO recording if your PC or Mac includes hardware encoding. However, if your system includes hardware encoding only via NVIDIA NVENC, then you will not be able to record if you are using an interlaced format (e.g. formats that include "i" such as 1080i59.94, 1080i60, etc.) as this is a limitation of NVIDIA's encoding capabilities. Only systems that include hardware encoding via Intel Quick Sync would support these formats.
As a workaround, go to Settings: Project Format and select a progressive project format (e.g. formats that include "p" such as 1080p30, 720p30). Another alternative is to go to Settings: Recording and switch the Default File Format to MJPEG, but this would create a larger, uncompressed file.