Because the custom column is a checkbox type field, you don't need to use an If() function to set the condition. If the conditions listed are met, the value in the custom column is automatically Yes (in other words, the checkbox is turned on).
Once you create this formula for the custom column of your report, you must set the filtering section to take advantage of it. Select Custom Column is and then type in the word YES.
Read more about Custom Formula columns here:
https://www.quickbase.com/help/custom_column_in_view.html