Yes. If your property is 35 acres or larger you may. If your property is less than 35 acres, you may not, until you have a primary residence on the property, or obtain a building permit for a primary residence.
Please note: An accessory structure may not be occupied as a dwelling. Even though the accessory structure may or may not need a building permit, it must meet the current setbacks associated with your zone district and the current building code.