Can a custom form be configured to be mandatory?

This feature is available on the Master payroll edition.
Yes. Custom forms can be configured to be mandatory by enabling the "Required" checkbox on the Custom Form Configuration screen.
Navigate > Configuration > Basic Settings > Custom Screens & Fields > Custom Form Configuration