Should you try your hardest? Generally the answer is no. Most things in life we could have put more effort into, but we need and want to sleep, eat, ect.
Over at Underlying Logic, Erik Simpson discusses the role of optimal effort and students. The discussion began with Usain Bolt, the Olympic sprinter who eased up at the end of his raise. In his teaching Erik has found that "I slowly came to realize that many of my students were choosing to incur penalties consistently so that I never got a chance to judge their best work in a straightforward way. That was the point. If you never try your hardest, nobody can ever find your limits. "
Erik makes more of psychological argument that we feel worse if we try and fail, then we fail without effort. But economist generally assume effort is costly. In the case of Bolt, that might not be, but a student would have to work longer hours. Additionally, hours worked not only have decreasing productivity the more you work (diminishing returns) and the leisure forgone when you are already working hard is extremely valuable. Try working another hour if you have worked 20 in a row, compared to the first hour worked.
There is a large economic literature on signaling models related to trying and failing. Simple example. Imagine I'm your boss, and I assign you a task say you plant carrots on my farm. I have too many workers to watch you plant carrots, so I pay you based on the number of carrots produced. Output is based not only on your effort, but also random forces (rain, rabbits, ect.). Depending on the payoffs to a good harvest or a bad harvest, your ability, and the impact of outside forces you may choose to put in a lot of effort or not.
In some sense students have only a vague idea of how an extra hour of studying will pay off both in terms of their knowledge and their grades. Part of their grade is their effort, and part is random forces (did they study the right material, is the prof in a bad mood when they grade the exam).
To optimally get the best effort generally the models conclude that high effort must be more likely to yield the good result, which is then rewarded.
So assign tough assignments, where high effort pays off.