Yes, custom formula fields can be used in workflow rules.

However, the rules for triggering workflow have not changed.  Workflow rules are only evaluated when a record is created or updated. 

This means that you cannot use a custom formula as a way to trigger a workflow rule based on the passage of time.  For example, if you create a custom formula field on Case called "Days Open", you can't create a workflow rule that automatically triggers when Days Open > 10.  Well, to be precise, you you can create the workflow rule, but it will only be evaluated when the Case is updated by an end user or through the API.

Time-based workflow is on the Customforce product roadmap.