Custom formulas were not intended as a solution for data validation, and we have plans to provide robust field and cross-field validation capabilities in the future.

However, if you are anxious for field validation logic today, it is possible to implement a workaround with a custom formula field, a custom report, and/or workflow.

1. Create a custom formula field with a return type of Text
2. Use an IF function to evaluate one or more validation conditions.  If an error condition is found, return a textual description of the error.  If there are no errors, return an empty string ("").
3. Create a custom report that lists data records where the custom formula field is non-blank.
4. Alternatively, you can create a workflow rule that creates a task or sends an email notification when the custom formula field is non-blank.

For example, here is an Opportunity formula that checks if a custom field Renewal Expiration Date is after the Close Date. 

IF ({!Expiration_Date__c} > {!CloseDate}, "Error: renewal expiration date cannot be after close date", "")

>The major limitation with this approach is that it does not prevent the user from saving invalid data.  Rather, it enables you to detect invalid data after it has been saved, so that it can be corrected.