Normally, you should use the latest version of Joomla! for a new site unless you have a specific reason not to. The latest version can be downloaded from the Joomla! download site. If you are running an earlier release of the current version (for example, you are running 1.5.2 and the current version is 1.5.5), you should upgrade to the current version if you experience any problems. Each minor release fixes a number of bugs, so it is always good practice to upgrade to the latest minor version as a first step in troubleshooting a problem.
If you have a site that is working well on an older Joomla! version, you do not need to upgrade the site. At the time of this writing, both Joomla! version 1.0 and 1.5 are supported, although all new development effort is going into version 1.5. Version 1.6 is planned for release in a few months time.
Many Joomla! Extensions have been released for version 1.5, with more being offered each week. However, there may be specific cases where an Extension is available for version 1.0 and not available for version 1.5, and this could be a reason to use version 1.0.