Hong Kong was once thought of as a cheap destination, but that is not really the case anymore. Compared to most other large cities in Asia it’s fairly expensive, though there are definitely a fair amount of cheap options for those on backpacker budgets.
Hotels in particular can be pricey in Hong Kong, especially if you want to stay on the island itself. Staying in the southern part of Kowloon will save you some money while keeping you in the center of the action. Hostels in Hong Kong offer good bargains as well, and private rooms are often quite a bit cheaper than 2-star hotels, but with similar services.
As long as you don’t plan on going to Disneyland or the other super-commercial options then the attractions in Hong Kong are reasonably priced. More good news is that food can be quite cheap if you stick more to street stalls and other places where locals eat, but if you go to places that cater mostly to Western tourists you can spend a bundle.