Here's a quick script that will prevent large attachments from being added to catalogue items in the Service Portal

function onSubmit() {

    var attachments = this.angular.element("#sc_cat_item").scope().attachments;
    var submitForm = true;
    attachments.forEach(function(attachment){
        var sizeDetails = attachment.size_bytes.split(' ');
        if(parseInt(sizeDetails[0])>10 && sizeDetails[1]=="MB"){
            alert(attachment.file_name +' is too big. Please remove and upload a smaller file');
            submitForm = false;
        }
    });

    if(submitForm){return true;}else{return false;}   
}

Just add this as an on submit catalog client script (being sure to check that "isolate script" is disabled AFTER you've saved your script)

The advantage of this approach is you can restrict some but not all catalog items (ie, if you have a heavily used catalog item, you might want to stop large uploads, but there may be other areas where you want to allow larger files)

Have fun